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Learning  Objectives 


Understand  the  limitations  of  traditional  risk  management  approaches 
for  today’s  complex,  multi-organizational,  system-of-system  programs. 

Understand  how  current  program  conditions  can  be  used  to  estimate  the 
program’s  current  momentum  towards  success. 

Learn  how  to  use  the  Mission  Diagnostic  to  evaluate  a  program’s  key 
drivers  of  success  and  failure  and  determine  it’s  current  potential  for 
success. 


Software  Engineering  Institute 


Carnegie  Mellon 


New  Directions  in  Risk 
Chris  Alberts  &  Audrey  Dorofee 

©  2009  Carnegie  Mellon  University 


4 


CURRENT  PROBLEM  SPACE 
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Widespread  Use  of  Risk  Management 
Approaches 


Most  programs  or  organizations  implement  some  type  of  risk 
management  approach. 

•  Risk  management  plan 

•  Processes 

•  Tools 

However,  preventable  failures  continue  to  occur. 

•  Uneven  and  inconsistent  application  of  risk-management  practice 

•  Significant  gaps  in  risk-management  practice 

•  Ineffective  integration  of  risk-management  practice 


Software  Engineering  Institute 


Carnegie  Mellon 


New  Directions  in  Risk 
Chris  Alberts  &  Audrey  Dorofee 

©  2009  Carnegie  Mellon  University 


6 


Increasing  Complexity 


Managers  are  overseeing  increasingly  complex  projects,  programs,  and 
operational  processes. 

•  Multiple  models,  frameworks,  and  standards 

•  Multiple  points  of  management  control 

•  Complex,  distributed  support  technologies 

•  A  variety  of  management  techniques  (project,  security,  financial,  technology, 
etc.) 

•  Complex  tasks 

•  Multiple  detailed  status  reports 
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Single  Point  of  Management  Control 


Many  management  approaches 
are  based  on  a  single  point  of 
management  control. 


Organization  A 


Single  Program 


Organization  A 


Framework  1 


Category  1 

Category  2 

Category  3 

Attribute  1.1 
Attribute  1.2 
Attribute  1.3 

Attribute  2.1 
Attribute  2.2 
Attribute  2.3 
Attribute  2.4 

Attribute  3.1 
Attribute  3.2 

Single  IT  System 


Single  Management  Framework 
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Distributed  Management  Environments 


A  distributed  management 
environment  comprises  multiple, 
points  of  management  control. 


Organization  A  \~ 

-[a2^ 

1 - ^ 

Organization  B 

— B1  ^  B5  ^  B6  ^ 

Multiple  Organizations 


Organization  B 


Framework  1  Framework  3 


Category  1  Category  2  Category  3  Category  1  Category  2  Category  3  Category  4  Category  5 


Attribute  4.1  Attribute  5.1 

Attribute  4.2  Attribute  5.2 

Attribute  5.3 
Attribute  5.4 
Attribute  5.5 


Framework  2 

I 

Category  1  Category  2 


Category  1.1  Category  1.2  Category  2.1  Category  2.2  Category  2.3 


Attribute  1.1  Attribute  2.1  Attribute  3.1 

Attribute  1.2  Attribute  2.2  Attribute  3.2 

Attribute  1.3  Attribute  2.3 

Attribute  2.4 


Attribute  1.1  Attribute  2.1  Attribute  3.1 

Attribute  1.2  Attribute  2.2  Attribute  3.2 

Attribute  1.3  Attribute  3.3 


Attribute  1.1.1  Attribute  1.2.1  Attribute  2.1.1  Attribute  2.2.1  Attribute  2.3.1 

Attribute  1.1.2  Attribute  1.2.2  Attribute  2.1.2  Attribute  2.2.2  Attribute  2.3.2 

Attribute  1.1.3  Attribute  2.2.3  Attribute  2.3.3 

Attribute  2.3.4 
Attribute  2.3.5 


Multiple  Systems 


Multiple  Frameworks 
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Changing  Management  Environment 


Old  Environment 

New  Environment 

•  Centralized  knowledge, 

•  Distributed  knowledge. 

experience,  and  expertise 

experience,  and  expertise 

•  Single  point  of  management 

•  Multiple  points  of  management 

control  (few  decision  makers) 

control  (many  decision  makers) 

•  Command  and  control 

•  Communication  and  coordination 

Software  Engineering  Institute 


Carnegie  Mellon 


New  Directions  in  Risk 

Chris  Alberts  &  Audrey  Dorofee  10 

©  2009  Carnegie  Mellon  University 


Management  Challenges 


Implementing  an  integrated  management  approach 

•  Synthesizing  the  results  of  point  solutions 

•  Focusing  on  operational  success  across  the  life  cycle  and  supply  chain 

•  Strategically  allocating  resources  based  on  greatest  need 

•  Striking  the  proper  balance  between  risk  and  opportunity 

Coordinating  management  efforts  in  distributed  environments 
Balancing  strategic  objectives  (mission)  and  tactical  objectives  (local) 
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Issues  with  Current  Risk  Management  Solutions 


Prevalence  of  point  solutions 

•  Specific  point  in  the  life-cycle 

•  Narrow  range  of  threats 

Inability  to  scale  point  solutions  to  distributed  management 
environments 

Lack  of  risk-management  solutions  specifically  designed  for  distributed 
management  environments  (e.g.,  system-of-system  environments) 
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Risk  Management 

A  REVIEW 
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Exercise  One 


Refer  to  Tutorial  Workbook,  Exercise  #1 

1 .  Read  the  Scenario 

2.  Consider: 

•  What  led  to  the  program’s  failure? 

•  Who  should  have  been  responsible  for  resolving 
these  issues  and  preventing  this  failure? 
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What  Is  Risk? 


Risk  is  the  likelihood  of  loss. 


Risk  requires  the  following  conditions: 

•  A  potential  loss 

•  Likelihood 

•  Choice 
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Risk  Perspectives 


Speculative  Perspective 

Provides  the  potential  for  gain  as  well  as  the  potential  for  loss 

Brings  the  potential  to  improve  the  current  situation  relative  to 
the  status  quo 


Hazard  Perspective 

Provides  no  opportunity  to  improve 
upon  the  current  situation 

Brings  only  the  potential  for  loss 
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Risk  in  Software-Intensive  Systems 


When  developing  or  operating  software-intensive  systems,  risk  is 
traditionally  viewed: 

•  From  a  hazard  perspective 

•  As  a  potential  obstacle  that  can  interfere  with  progress 
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The  Traditional  View  of  Risk:  Focus  on  Threats 


A  threat  is  a  circumstance  with  the  potential  to  cause  harm  or  loss. 

•  Conditions  with  negative  consequences 

•  Events  with  negative  consequences 


Context 


/ 

1 

/ 

Success 

/ 

H 

Threats - ► 

Execution 

3 

V 

4 

Failure 

\ 

5 

Range  of  Potential  Outcomes 

V 

J 
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Risk  Statement 


Threat 

A  phrase  or  sentence  that  briefly  describes  the  circumstances  and 
situations  that  are  causing  concern,  doubt,  anxiety,  or  uncertainty 

Consequence 

A  phrase  or  sentence  that  describes  the  negative  outcome(s)  resulting 
from  the  current  conditions 

Risk  Statement 
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Example:  Risk  Statements 


Threat  Consequence 

The  program  could  fail  to  achieve 

Staffing  levels  are  insufficient  its  product,  cost,  and  schedule 

objectives 

If  our  subcontractor  is  late  in  getting  Then  the  program’s  schedule  will 
its  modules  completed  on  time  likely  slip 


The  first  risk  statennent  is  a  condition-consequence  statement,  which  is 
effective  for  articulating  risks  triggered  by  current  conditions. 

The  second  risk  statement  is  an  if-then  statement,  which  is  effective  for 
articulating  risks  triggered  by  the  occurrence  of  potential  events. 
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Types  of  Risk 


•  Enterprise  risk 

•  Program  risk 

•  Business-process  risk 

•  Product  risk 

•  Operational  risk 

•  Design  risk 

•  Integration  risk 

•  IT  risk 

Contracting/ 

•  Architecture  risk 

•  Performance  risk 

•  Information  assurance  risk 

acquisition  risk 

•  COTS  risk 

•  User  acceptance  risk 

•  Security  risk 

Contracting 

Development 

^  Transition  to 

Operations 

I 

Operations  and 
Maintenance 

Many  different  types  of  risk  are  managed  across  the  life  cycle. 

Threat-driven  approaches  are  most  commonly  used 

to  manage  these  risks. 
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What  Is  Risk  Management? 


In  a  systems  context,  risk  management  is  traditionally  viewed  as  a 
proactive,  disciplined  approach  for 


•  Assessing  what  can  go  wrong — risks 
caused  by  a  range  of  threats 

•  Determining  which  risks  are  important 
to  address 

•  Implementing  actions  to  deal  with 
the  highest  priority  risks 
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A  Different  Perspective 

KEY  CONCEPTS 
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Multiple  Risk  Management  Approaches 


Focus  of  this  tutorial 
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Objective-Driven  Risk  Management  -  2 


A  proactive,  disciplined  approach  for 

•  Identifying  the  key  objectives  that  must  be  achieved 

•  Establishing  what  factors,  or  drivers, 
can  influence  the  outcome 


Determining  which  drivers  are  putting 
key  objectives  at  risk 

Assessing  the  probability  and  severity 
of  each  risk 

Developing  mitigation 
approaches  for  each  risk 

Implementing  and  tracking  implementation 
plans 
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Two  Types  of  Risk  Management 


Threat-Driven  Risk  Management 
(Risk  to  Execution) 

Objective-Driven  Risk  Management 
(Risk  to  Objectives) 

Key  Questions 

Key  Questions 

1 .  What  threats  (i.e.,  potential 

1 .  What  key  objectives  must  be 

hazards)  can  interfere  with 

achieved? 

execution? 

2.  What  factors,  or  drivers,  can 

2.  How  likely  is  each  threat? 

influence  the  outcome? 

3.  What  is  the  severity  of  impact 

3.  Which  drivers  are  putting  the  key 

for  each  threat? 

objectives  at  risk? 

4.  What  is  the  probability  and 
severity  of  each  risk? 

Tacticai  focus 

Systemic  focus 

Bottom-up  analysis 

Top-down  anaiysis 
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Managing  Risk  in  Distributed  Environments 


Objective-driven  risk  management  is  a  structured  approach  for 
assessing  and  managing  in  distributed  environments. 

•  Multiple  teams,  groups,  or  organizations  working  toward  common  objectives 
{current  focus) 

•  Systems  of  systems  and  networked  systems 

•  Management  of  multiple  models,  standards,  &  frameworks 

•  Others 
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Mission-Oriented  Success  Analysis  and 
Improvement  Criteria  (MOSAIC) 

What 

A  suite  of  methods  that  enable  objective-driven  risk  management 


Benefits 

Enables  continuous  management  of  risk  to 
objectives 

Applicable  across  all  life-cycle  phases 

Designed  for  distributed  management 
environments 

Provides  a  means  of  analyzing  risk  in  relation  to 
management  models,  frameworks,  and 
standards 


— New  Directions  in  Risk 
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MOSAIC:  Overarching  Goal 


To  establish  and  maintain  confidence  that  a  software-intensive  system 
or  system  of  systems  will  achieve  its  key  operational  objectives 

•  Support  of  operations 

•  Functionality 

•  Performance 

•  Reliability 

•  Interoperability 

•  Information  assurance 

•  Usability  and  maintainability 

•  Others 
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Risk  as  an  Integrating  Theme 


The  risk  to  objectives  is  used  to 
create  a  singie,  integrated  view  of 
the  current  state  across  muitipie, 
disparate  entities. 


Organization  A  ^ 

1 — ^ 

Organization  B 

Organization  C 


Multiple  Organizations 


Organization  B 


Framework  1 


- 1 - 1 

Category  2  Category  3 


I - 

Category  1 


Attribute  1.1 
Attribute  1.2 
Attribute  1.3 


Attribute  2.1 
Attribute  2.2 
Attribute  2.3 
Attribute  2.4 


Attribute  3.1 
Attribute  3.2 


Framework  3 


Category  1 


Attribute  1.1 
Attribute  1.2 
Attribute  1.3 


I 

Category  2 


Attribute  2.1 
Attribute  2.2 


Category  3 


Attribute  3.1 
Attribute  3.2 
Attribute  3.3 


I 

Category  4 


Attribute  4.1 
Attribute  4.2 


Category  5 


Attribute  5.1 
Attribute  5.2 
Attribute  5.3 
Attribute  5.4 
Attribute  5.5 


Category  1 


Category  1.1  Category  1.2 


Framework  2 


Category  2 


Category  2.1  Category  2.2 


1 


Category  2.3 


Attribute  1.1.1  Attribute  1.2.1 

Attribute  1.1.2  Attribute  1.2.2 

Attribute  1.1.3 


Attribute  2.1.1 
Attribute  2.1.2 


Attribute  2.2.1 
Attribute  2.2.2 
Attribute  2.2.3 


Attribute  2.3.1 
Attribute  2.3.2 
Attribute  2.3.3 
Attribute  2.3.4 
Attribute  2.3.5 


Multiple  Systems 


Multiple  Frameworks 
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MOSAIC:  Establishing  and  Maintaining 
Confidence 


Context 


Success 


Failure 


Range  of  Potential  Outcomes 


Confidence  is  established  by  analyzing  the  effects  of 

•  Current  conditions  (with  positive  and  negative  consequences) 

•  Events  (with  positive  and  negative  consequences) 
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MOSAIC:  Managing  Outcomes  and  Events  - 1 


Integrated 
Management  • 


Outcome 

Management 


Event 

Management 


Probability  of  success 
Uncertainty  vaiue 


Sensitivity  to  events 


Achieving  key  objectives  requires 

1 .  Establishing  sufficient  momentum  toward  key  objectives 
(outcome  management) 

2.  Maintaining  momentum  by  managing  events  (event  management) 
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MOSAIC:  Managing  Outcomes  and  Events  -  2 


The  goal  of  outcome  management  is  to  maximize  the  overall  likelihood 
of 

•  Achieving  key  objectives 

•  Realizing  the  business/mission  opportunity 

The  goal  of  event  management  is  to 

•  Maximize  the  potential,  positive  consequences  of  events  (tactical 
opportunities) 

•  Minimize  the  potential,  negative 
consequences  of  events  (tactical  risks) 
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MOSAIC:  Focus  on  Key  Objectives  - 1 


An  objective  is  a  desired  outcome,  or  future  result. 

A  key  objective 

•  Is  a  vital  outcome  intended  to  be  achieved  in  the  future 

•  Provides  a  benchmark  against  which  success  will  be  judged 

A  set  of  key  objectives  define  the  mission,  or  picture  of  success,  for  a 
project  or  process. 
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MOSAIC:  Focus  on  Key  Objectives  -  2 


Key  objectives  typically  incorporate  multiple  perspectives,  including 
•  Program 

Operational  and  mission 
Business 
Enterprise 
Stakeholders 

Near-  and  long-term  views  of  success 

Others  HSUP 
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MOSAIC:  Focus  on  Key  Objectives  -  3 


Key  objectives  are  used  to  set  the 
scope  when  assessing  in  a 
distributed  environment. 


Multiple  Organizations 


itegory  1  Category  2  Category  3  Category  4  Category  5 


1  ^ 


Category  1  Category  2  Category  3 


ite1.1  Attribute  2.1  Attribute  3.1 

ite1.2  Attribute  2.2  Attribute  3.2 

ite1.3  Attribute  2.3 

Attribute  2A 


I - H — 

Category  Z1  Category  2.2 


> 


T 


Key 

Objectives 


Attribute  1.1.1  Attribute  1.2.1  Attribute  2.1.1  Attribute  2.2.1 

Attribute  1.1.2  Attribute  1.2.2  Attribute  2.1.2  Attribute  2.Z2 

Attribute  1.1.3  Attribute  2.2.3 


Attribute  Z3.1 
Attribute  Z3.2 
Attribute  Z3.3 
Attribute  Z3.4 
Attribute  Z3.5 


Multipie  Frameworks 
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Network  of  Objectives 


Key  Program 
Objectives 


Success  of  a  collaborative  venture  requires  ensuring  that  all  of  the  key 
objectives  within  the  network  are  aligned  and  on  track  for  success. 

MOSAIC  is  designed  to  manage  risk  across  a  network  of  objectives. 
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Drivers  of  Program  Success  (and  Failure) 


A  driver  is  a  key  factor  that  steers  a  program  towards  success  or  failure. 

Drivers  are  derived  from  key  objectives 

•  Drivers  are  contextual  -  specific  to  a  program. 

•  Many  drivers  are,  however,  common  to  most  programs. 


Key  Objectives 


Driver  1  Driver  2  Drivers  ...  Driver  N 
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Evaluating  Drivers 


Each  driver  is  evaluated  to  determine  how  it  is  currently  affecting  the 
outcome,  or  result. 

•  A  success  driver  guides  the  outcome  toward  the  desired  state  (i.e.,  key 
objectives). 

•  A  failure  driver  guides  the  outcome  away  from  the  desired  state,  creating 


risk  to  objectives. 

Key  Objectives 

Positive  Conditions  — =- - 

_ ^  Driver  1 

Objective  1 
Objective  2 

■ 

and  Potential  Events 

Driver  2 

Driver  3 

Negative  Conditions 

■ 

f 

■ 

and  Potential  Events 

■ 

■ 

X  ■ 

Driver  N  ^ 

Objective  M 
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Drivers  and  Momentum 


The  current  values  of  the  success  and  failure  drivers  indicate  how  much 
momentum  towards  success  the  program  has  and  how  resilient  it  is  to 
risks  and  unexpected  events. 
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Driver  Attributes 


Attribute 

Description 

Example 

Name 

Short,  1-3  word  identifier 

Task  Execution 

Success 

State 

Driver  acting  as  a  factor 
for  success  (in  statement 
form) 

Tasks  and  activities  are 
performed  effectively  and 
efficiently. 

Failure 

State 

Driver  acting  as  a  factor 
for  failure  (in  statement 
form) 

Tasks  and  activities  are  not 
performed  effectively  and 
efficiently. 

Category 

Each  driver  belongs  to 
one  of  six  categories 

Execution 

Software  Engineering  Institute 


Carnegie  Mellon 


New  Directions  in  Risk 

Chris  Alberts  &  Audrey  Dorofee  41 

©  2009  Carnegie  Meiion  University 


Driver  Categories 


Six  standard  categories  of  drivers  are  considered  when  assessing 
software  programs. 

Objectives 
Preparation 
Environment 
Execution 
Resilience 
Product 


i 
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Primary  Relationships  Among  the  Driver 
Categories 
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Example:  Drivers  for  Distributed  Software 
Programs 


Driver  Framework 


1 


Obiectives 

Environment 

Resilience 

1 .  P  rog  ra  m  0  bjecti ves 

4.  Organizational  Conditions 

5.  Compliance 

12.  Event  Management 

Preoaration 

Execution 

Product 

2.  Plan 

6. 

Task  Execution 

13. 

Requirements 

3.  Process 

7. 

Coordination 

14. 

Design  and  Architecture 

8. 

External  Interfaces 

15. 

System  Capability 

9. 

Information  Management 

16. 

System  Integration 

10. 

Technology 

17. 

Operational  Support 

11. 

Facilities  and  Equipment 

18. 

Adoption  Barriers 

19. 

Operational  Preparedness 

20. 

Certification  and  Accreditation 
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Exercise  Two 


Refer  to  Tutorial  Workbook,  Exercise  #2  and  the  Scenario  from 
Exercise  #1 


Consider  the  following  question: 

•  Which  failure  drivers  contributed  to  the  problems  experienced  by  the 
program? 
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Changing  Risk  Management  Paradigm 


Threat-Driven  Risk 
Management 

Objective-Driven  Risk 
Management 

•  Managing  hazards 

•  Achieving  key  objectives 

•  Centralized  management 

•  Distributed/collaborative  management 

•  Point  solutions 

•  Integrated  approach 

•  Tactical  focus 

•  Systemic  focus 

•  Bottom-up  analysis 

•  Top-down  analysis 
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THE  MISSION  DIAGNOSTIC 
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Mission  Diagnostics:  Foundation  for  MOSAIC 


The  Mission  Diagnostic  approach  provides  the  foundation  for  all 
MOSAIC  assessments. 

•  Identify  key  objectives 

•  Select  and  tailor  the  drivers 

•  Analyze  drivers 

•  Analyze  risk 
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Mission  Diagnostics 


A  class  of  driver-based  assessments  that  incorporate  a  basic  back-end 
analysis,  such  as 

•  Gap  analysis  (mission  gap  diagnostic) 

•  Risk  analysis  (mission  risk  diagnostic) 

•  Success  analysis  (mission  success  diagnostic) 

The  Mission  Diagnostic  in  this  tutorial  is  the  mission  success  diagnostic. 
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What  Is  a  Mission? 


The  term  mission  has  multiple  meanings,  depending  on  the  context  in 
which  it  is  used. 

For  example,  mission  is  used  to  describe  any  of  the  following: 

•  Purpose  of  an  organization 

•  Goals  of  a  specific  department  or  group  within  a  larger  organization 

•  Objectives  of  each  activity  in  a  work  process 

•  Function  of  each  technology  (e.g.,  a  software-intensive  system)  that  supports 
a  project  or  process 

•  Specific  result  being  pursued  when  executing  a  project  or  process 
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Mission  Diagnostic  Activities 
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The  Mission  Diagnostic 

ESTABLISHING  KEY  OBJECTIVES 
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Establishing  Key  Objectives 


Key  objectives  define  the  desired 
outcome  at  a  future  point  in  time. 


Key 
Objectives 


01 .  Key  Objective  1 
02.  Key  Objective  2 


Future  Point  in  Time 


ON.  Key  Objective  N 


t 


current 


^future 
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Example:  Key  Objective 


By  the  end  of  the  initial  deployment 
phase  (6  months),  the  payroll  application 
will  fully  support  operations  at  the  initial 
deployment  site. 


Key  ^ 

Objective 

The  payroll  application 
will  fully  support 
operations  at  the  initial 
deployment  site. 


■r 


6  Months 


t 


six  months 


t 


current 
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The  Mission  Diagnostic 

TAILOR  DRIVERS 
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Reminder:  What  Are  Drivers? 


A  driver  is  a  situation  or  circumstance  that  has  a  strong  influence  on  the 
outcome  or  result. 

•  A  success  dr/Ver  guides  the  outcome  toward  key  objectives. 

•  A  failure  c/r/Ver  guides  the  outcome  away  from  key  objectives. 


Positive  Conditions 
and  Potential  Events 


Negative  Conditions 
and  Potential  Events 


Driver  1 
Driver  2 
Driver  3 


Driver  n 


Excellent 


Potential  for  Success 
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Example:  Tailoring  Drivers  - 1 


1 .  Program  Objectives 

2.  Plan 

3.  Process 

4.  Organizational  Conditions 

5.  Compliance 

6.  Task  Execution 

7.  Coordination 

8.  External  Interfaces 

9.  Information  Management 

10.  Technology 

11.  Facilities  and  Equipment 

12.  Event  Management 


13.  Requirements 

14.  Design  and  Architecture 

15.  System  Capability 

16.  System  Integration 

17.  Operational  Support 

18.  Adoption  Barriers 

19.  Operational  Preparedness 

20.  Certification  and  Accreditation 
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Example:  Tailoring  Drivers  -  2 


Adjust  the  drivers  based  on  your  current  context  and  objectives,  for 
example 

•  Where  you  are  in  the  life  cycle 

•  Type  of  project,  such  as  new  development  vs.  maintenance 

•  Terminology 

Expand  drivers 

•  Plan  can  be  expanded  to  Plan,  Budget,  and  Schedule 
Collapse  drivers 

•  Operational  Support,  Adoption  Barriers,  and  Operational  Preparedness  could 
be  collapsed  into  Operations 

Add  drivers  as  needed  for  completely  new  aspects  or  new  types  of 
objectives 
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The  Mission  Diagnostic 

EVALUATE  DRIVERS 
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Collect  Information 


To  evaluate  a  driver,  you  need  information  from 

•  Program  personnel,  all  levels  and  groups 

•  Program  documentation 

•  Other  sources 

Gather  information  from 

•  Interviews 

•  Documentation  reviews 

•  Group  meetings  to  reach  consensus 
on  drivers 
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Example:  Driver  Question 


Question 

No 

Likely 

no 

Answer 

Equally 

likely 

Likely 

yes 

Yes 

Is  the  process  being  used  to  develop 
(and  deploy)  the  system  sufficient? 

□ 

□ 

□ 

□ 

□ 

Driver  questions  are  phrased  from  the  success  perspective. 
Probability  is  incorporated  into  the  range  of  answers  for  each  driver. 
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Example:  Driver  Value  Criteria 


Each  driver  is  evaluated  against  predefined  criteria. 

You  must  also  record  the  rationale  for  your  evaluation  of  each  driver. 


Measure 

Description 

Yes 

The  answer  is  almost  certainly  “yes.”  Almost  no  uncertainty  exists.  There  is  little 
or  no  probability  that  the  answer  could  be  “no.”  (~  >  95%  probability  of  yes) 

Likely  yes 

The  answer  is  most  likely  “yes.”  There  is  some  chance  that  the  answer  could  be 
“no.”  (~  75%  probability  of  yes) 

niie  answer  is  just  as  likely  to  be  “yes”  or  “no.” 

(~  50%  probability  of  yes) 

Likely  no 

The  answer  is  most  likely  “no.”  There  is  some  chance  that  the  answer  could  be 
“yes.”  (~  25%  probability  of  yes) 

No 

The  answer  is  almost  certainly  “no.”  Almost  no  uncertainty  exists.  There  is  little  or 
no  probability  that  the  answer  could  be  “yes.”  (~  <  5%  probability  of  yes) 
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Example:  Evaluating  Drivers 


Question  Answer 


No 

Likely 

no 

Equally 

likely 

Likely 

yes 

Yes 

1 .  Is  the  process  being  used  to  develop 
(and  deploy)  the  system  sufficient? 

□ 

X 

□ 

□ 

□ 

Each  driver  is  evaiuated  based  on  the  data  that  have  been  coliected. 
The  rationaie  for  selecting  an  answer  is  recorded. 
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Example:  Rationale  for  Driver  Value  - 1 


Driver 

Value 

1 .  Is  the  process  being  used  to  develop 
(and  deploy)  the  system  sufficient? 

Equally  Likely  Yes  or 
No 

Rationale 

+  Previous  programs  have  a  90%  history  of  deiivering  on-time  with 
required  functionaiity. 

+  The  iead  engineers  are  skiiled  at  adapting  to  new  processes. 

-  This  program  required  a  significant  change  in  our  standard  processes. 
There  was  no  new  training  created  for  the  new  processes. 
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Example:  Rationale  for  Driver  Value  -  2 


Rationale  (cont) 

-  QA  did  not  have  a  chance  to  review  the  new  and  revised  processes 
before  they  were  put  into  practice. 

-  The  person  who  developed  the  new  processes  quit  last  week. 

-  There  are  a  lot  of  brand  new  programmers  (45%). 
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The  Mission  Diagnostic 

RISK  PROFILE  OR  POTENTIAL 
FOR  SUCCESS 
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Example:  Driver  Profile  - 1 


Programmatic  Drivers 


Product  Drivers 


<u 


A  simple  analysis  provides  insight  into  the  potential  for  success. 
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Mission  Diagnostic  Results:  Driver  Profile  -  2 


Driver  Framework 


Objectives 

Environment 

Resilience 

■  1 .  Program  Objectives 

■  4.  Organizational  Conditions 

■  5.  Compliance 

□  12.  Event  Management 

Preoaration 

Execution 

Product 

□  2.  Plan 

□ 

6. 

Task  Execution 

H  13. 

Requirements 

I  I  3.  Process 

■ 

7. 

Coordination 

■  14. 

Design  and  Architecture 

□ 

8. 

External  Interfaces 

■  15. 

System  Capability 

□ 

9. 

Information  Management 

■  16. 

System  Integration 

■ 

10. 

Technology 

□  17. 

Operational  Support 

■ 

11. 

Facilities  and  Equipment 

□  18. 

Adoption  Barriers 

@  19. 

Operational  Preparedness 

■  20. 

Certification  and  Accreditation 

A  driver  profile  can  also  present  risks  in  relation  to  the 

driver  framework. 
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Mission  Diagnostic  Results:  Traditional  “Risk 
List" 


1 .  Project  objectives  are  unrealistic  or 
unachievable. 

14.  The  project  does  not  have  sufficient 
capacity  and  capability  to  identify  or 
manage  unpredictable  events  and 
changing  circumstances. 

7.  The  project  does  not  comply  with  all 
relevant  policies,  laws,  and  regulations. 


Risk 

Probability 


High 


Medium 


Medium 


Risk 

Severity 


Severe 


Risk 

Exposure 


High 


Severe 


Medium 


Low 


Minimum 


A  risk  profile  can  be  a  list  of  current  risks  (impact  to  key  objectives  is 

implied)  derived  from  drivers. 
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Potential  for  Success 


The  potential  for  success  is  the  likelihood  that  a  key  objective  will  be 
achieved  (also  called  the  probability  of  success). 

The  current  potential  for  success  is  an  indicator  of  mission  risk  and 
opportunity. 

•  A  high  potential  for  success  is  an  indicator  of  mission  opportunity. 

•  A  low  potential  for  success  is  an  indicator  of  mission  risk. 

An  analysis  of  present  conditions  (as  represented  by  drivers  of  success) 
is  used  to  establish  the  current  potential  for  success. 

•  Simple  aggregation  of  driver  values 

•  Weighted  aggregation  of  driver  values 

•  Mean  or  median  driver  value 

•  Rule-based  algorithms 
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Example:  Basic  Success  Profile 


0) 

o 

o 

3 

(O 

o 


<5 

c 

0) 

o 

Q. 


Excellent 

High 

Medium 

Low 

Minimal 


Success  Threshold 


_ Current  Potential  for  Success 

E 


A  basic  success  profile  depicts  the  current  potential  for  success  in  relation 
to  the  success  threshold. 

The  success  threshold  defines  the  desired,  or  target,  potential  for  success 
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Example:  Success  Criteria 


Each  key  objective  is  analyzed  in  relation  to  a  set  of  success  criteria  to 

determine  its  current  potential  for  success. 


Measure 

Description 

Excellent 

Current  conditions  are  extremely  favorable  for  a  successful  outcome. 

(~  >  95%  chance  of  success) 

High 

Current  conditions  are  favorable  for  a  successful  outcome. 

75%  chance  of  success) 

Current  conditions  are  mixed,  making  success  and  failure  equally  likely. 

50%  chance  of  success) 

Low 

Current  conditions  are  not  favorable  for  a  successful  outcome. 

25%  chance  of  success) 

Minimal 

Current  conditions  are  extremely  unfavorable  for  a  successful  outcome. 

<  5%  chance  of  success) 
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Example:  Objective-based  Potential  for  Success 


Excellent 


High 

Medium 


Low 


Minimal 


Potential  for  Success 


An  analysis  of  drivers  is  used  to  determine  the  current 
potential  for  success  for  each  objective. 
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Example:  Success  Potential  of  a  Key  Objective 


By  the  end  of  the  initial  deployment  phase  (6  months),  the  payroll 
application  will  fully  support  operations  at  the  initial  deployment  site. 

Current  Potential  for  Success 

Current  likelihood  of  achieving  this  scenario  is  Low 


Rationale 

•  System  functionality  was  cut  to  meet  the  deployment  schedule  at  the  initial 
deployment  site. 

•  The  contractor  developing  the  payroll  application  has  not  been  meeting  its 
milestones. 

•  The  integration  task  is  more  complicated  than  usual. 

-  The  integration  schedule  is  shorter  than  usual. 

-  The  infrastructure  is  dynamic  and  evolving. 

-  No  one  is  managing  the  common  enterprise  infrastructure. 

•  Changes  in  senior  management  could  affect  the  ability  to  resolve  infrastructure 
issues. 
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The  Mission  Diagnostic 

NEXT  STEPS 
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Next  Steps 


Determine  what  areas  need 

•  Further  investigation 

•  Improvement 

If  further  investigation  is  needed 

•  Gather  additional  information  to  clarify  uncertainties 

•  Continue  decomposing  drivers  to  get  at  deeper  issues 

•  Chose  alternate  methods  to  analyze  the  situation 

If  improvement  is  needed 

•  Determine  causes  of  weaknesses 

•  Develop  and  implement  improvement  plans 

•  Re-evaluate 
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The  Mission  Diagnostic 

YOUR  PROGRAM 
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Exercise  Three 


Refer  to  Tutorial  Workbook,  Exercise  #3 

1 .  Select  a  program,  project,  or  process  with  which  you  are 
knowledgeable. 

2.  Evaluate  it  using  the  set  of  drivers  provided  in  the  Workbook. 

3.  Sketch  your  risk  profile. 


Consider: 

•  Are  there  some  drivers  for  which  you  need  more  information? 

•  Where  would  you  get  that  information? 
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BEYOND  THE  BASIC  MISSION 
DIAGNOSTIC 
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Expanding  the  Mission  Diagnostic 


There  are  many  ways  to  expand  the  Mission  Diagnostic. 


Two  will  be  briefly  introduced: 

•  Dealing  with  uncertainty 

•  Handling  events 
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Uncertainty  about  Current  Conditions 


Uncertainty  is  defined  as  having  doubt  or  being  unsure  of  something. 

As  you  analyze  information,  one  or  more  of  the  following  will  likely  be 
true: 

•  Certain  information  is  not  available  or  is  unknown. 

•  You  do  not  trust  certain  information  based  on  its  source. 

•  Some  information  is  based  on  people’s  assumptions  or  opinions,  which  might 
prove  to  be  incorrect. 

Some  uncertainty  will  be  associated  with  the  potential  for  success  based 
on  having  doubts  about  or  being  unsure  of  current  conditions  (i.e.,  driver 
values). 
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Changes  in  Uncertainty  Over  Time 


Many  uncertainties  related  to  current  conditions  can  be  resolved. 

•  More  information  becomes  available  or  known. 

•  Information  from  untrusted  sources  can  be  verified. 

•  Assumptions  can  be  tested  and  proved  to  be  correct  or  incorrect. 

You  will  almost  always  have  some  degree  of  uncertainty  related  to 
current  conditions. 

•  You  will  not  be  able  to  resolve  all  uncertainties. 

•  Information  will  always  be  imperfect. 

•  Changing  conditions  will  produce  new  uncertainties. 
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Example:  Uncertain  ty  in  Drivers 


Project  Drivers 
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83 


Events 


An  event  is  an  unpredictable  occurrence  that  changes  the  current  state 
(i.e.,  status  quo). 

Events  can  have  a  positive  or  negative  effect  on  the  outcome. 

•  A  decrease  in  funding  would  likely  produce  a  negative  consequence  that 
might  adversely  affect  a  project’s  outcome. 

•  An  increase  in  funding  would  likely  produce  a  positive  consequence  that 
might  put  a  project  in  better  position  for  success. 

A  sensitivity  analysis  examines  an  event’s  likely  effect  on  the  potential 
for  success 

•  Increase  in  the  potential  for  success  resulting  from  the  occurrence  of  an  event 
(i.e.,  tactical  opportunity) 

•  Decrease  in  the  potential  for  success  resulting  from  the  occurrence  of  an 
event  (i.e.,  tactical  risk  ) 


Software  Engineering  Institute 


Carnegie  Mellon 


New  Directions  in  Risk 

Chris  Alberts  &  Audrey  Dorofee  84 

©  2009  Carnegie  Mellon  University 


Event  Identification 


Relevant  events  are  identified  based  on 

•  Uncertainties 

Resolution  of  uncertainties  can  change  current  conditions  and  affect  the 
potential  for  success. 

•  Vuinerabiiities 

Inherent  weaknesses  can  expose  a  program  to  the  effects  of  unpredictable 
occurrences. 


Software  Engineering  Institute 


Carnegie  Mellon 


New  Directions  in  Risk 

Chris  Alberts  &  Audrey  Dorofee  85 

©  2009  Carnegie  Mellon  University 


Example:  Events  Resulting  from  an  Uncertainty 


The  status  of  application 
development  by  the 
contractor  is  uncertain. 


Event  1 

The  contractor  delivers  the 
application  on  time 


Event  2 

The  contractor  does  not 
deliver  the  application  on  time 
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Example:  Events  Resulting  from  an  Uncertainty 
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Example:  If  the  contractor  delivers  the 
application  on  time  (Event  1) 
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If  the  contractor  delivers  the  application  on  time,  the  values  of  all  drivers 

will  remain  at  their  current  levels. 


The  potential  for  success  will  remain  Low. 
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Example:  If  the  contractor  does  not  deliver  the 
application  on  time  (Event  2) 


If  the  contractor  does  not  deliver  the  application  on  time,  the  values  of 
the  drivers  for  plans  and  system  integration  will  decrease. 

The  potential  for  success  will  drop  to  Minimal. 
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Example:  Event  Resulting  from  a  Vulnerability 


Event  5 

The  deployment  schedule  Is 
compressed  further 
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Example:  Event  Resulting  from  a  Vulnerability 


the  deployment  schedule  is 
compressed  further 


Then  the  project’s  potential  for 
success  will  fall  to  minimal 


Context 

Senior  management  has  had  a  history  of  overriding  the  decisions  of  the 
project’s  management  team.  Promises  made  by  the  CIO  had  already 
shortened  the  development  schedule. 

The  integration  issues  will  be  difficult  to  resolve.  These  issues  make  the 
project  especially  vulnerable  to  further  schedule  changes. 
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Example:  If  the  deployment  schedule  is 
compressed  (Event  5) 


If  the  deployment  schedule  is  compressed  any  further,  the  drivers  for 
plans,  system  integration,  adoption  barriers,  and  operational 

preparedness  will  be  affected. 

The  potential  for  success  will  drop  to  Minimal. 


Software  Engineering  institute 


Carnegie  Mellon 


New  Directions  in  Risk 

Chris  Alberts  &  Audrey  Dorofee  92 

©  2009  Carnegie  Mellon  University 


Example:  Success  Profile  with  Event  Sensitivity 
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The  consequences  of  events  can  be  added  to  the 

basic  success  profile. 

This  success  profile  includes  one  event  that  increases  the 
potential  for  success  -  a  tactical  opportunity. 
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Summary  of  Key  Points  - 1 


The  paradigm  for  managing  software  programs  is  changing. 

•  Increased  complexity 

•  Distributed  knowledge,  experience,  and  expertise 

•  Multiple  points  of  management  control 

•  Focus  on  communication  and  coordination 

Objective-driven  risk  management 

•  Is  a  structured  approach  for  assessing  and  managing  in  distributed 
environments. 

-  Systemic  focus 

-  Top-down  analysis 

•  Uses  the  risk  to  objectives  to  create  a  single,  integrated  view  of  the  current 
state  across  multiple,  disparate  entities 
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Summary  of  Key  Points  -  2 


MOSAIC 

•  Is  a  suite  of  methods  that  enable  objective-driven  risk  management 

•  Comprises  a  range  of  assessments 

-  Basic 

-  Intermediate 

-  Advanced 

The  Mission  Diagnostic 

•  Provides  a  time-efficient  means  of  assessing  risks  to  program  objectives 

•  Focuses  on  a  set  of  key  drivers 
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Example:  Drivers  for  Distributed  Software 
Programs 


Driver  Framework 


Objectives 

Environment 

Resilience 

1 .  Program  Objectives 

4.  Organizational  Conditions 

5.  Compliance 

12.  Event  Management 

Preoaration 

Execution 

Product 

2.  Plan 

6. 

Task  Execution 
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Requirements 

3.  Process 

7. 

Coordination 

14. 

Design  and  Architecture 

8. 

External  Interfaces 

15. 

System  Capability 

9. 

Information  Management 
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System  Integration 

10. 

Technology 

17. 

Operational  Support 

11. 

Facilities  and  Equipment 

18. 

Adoption  Barriers 

19. 

Operational  Preparedness 

20. 

Certification  and  Accreditation 
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For  Additional  Information 


Christopher  Alberts 
Email:  cja@sei.cmu.edu 
Phone:412-268-3045 
Fax:  412-268-5758 


Audrey  Dorofee 
Email:  ajd@sei.cmu.edu 
Phone:412-268-6396 
Fax:  412-268-5758 


WWW  http://www.sei.cmu.edu/msce/index.html 

U.S.  mail  Software  Engineering  Institute 

Carnegie  Mellon  University 
Pittsburgh,  PA  15213-3890 


For  updated  slides 

http://www.sei.cmu.edu/msce/presentations.html 
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NO  WARRANTY 


THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE  ENGINEERING  INSTITUTE  MATERIAL  IS  FURNISHED 
ON  AN  “AS-IS"  BASIS.  CARNEGIE  MELLON  UNIVERSITY  MAKES  NO  WARRANTIES  OF  ANY  KIND,  EITHER 
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PURPOSE  OR  MERCHANTABILITY,  EXCLUSIVITY,  OR  RESULTS  OBTAINED  FROM  USE  OF  THE  MATERIAL. 
CARNEGIE  MELLON  UNIVERSITY  DOES  NOT  MAKE  ANY  WARRANTY  OF  ANY  KIND  WITH  RESPECT  TO 
FREEDOM  FROM  PATENT,  TRADEMARK,  OR  COPYRIGHT  INFRINGEMENT. 


Use  of  any  trademarks  in  this  presentation  is  not  intended  in  any  way  to  infringe  on  the  rights  of  the  trademark  holder. 

Internal  use.  Permission  to  reproduce  this  presentation  in  its  entirety  with  no  modifications  is  granted. 
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